Skip to content

feat: Qwen3-TTS 0.6B voice clone - #14

Merged
wavekat-eason merged 26 commits into
mainfrom
feat/qwen3-tts-0.6b-voice-clone
Apr 17, 2026
Merged

feat: Qwen3-TTS 0.6B voice clone#14
wavekat-eason merged 26 commits into
mainfrom
feat/qwen3-tts-0.6b-voice-clone

Conversation

@wavekat-eason

@wavekat-eason wavekat-eason commented Apr 17, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Add Qwen3TtsClone Rust backend for 0.6B voice-clone inference (speaker encoder → tokenizer encoder → LLM → code-to-wave)
  • Add ONNX export scripts for speaker and tokenizer encoders, plus end-to-end clone generation tool
  • Unify CI ONNX export workflow with a variant selector (base / clone)
  • Include reference WAV fixture, docs, and README updates

Test plan

  • make test-qwen3 passes
  • cargo run --example synthesize_clone --features qwen3-tts produces audio
  • ONNX export workflow runs for both base and clone variants

🤖 Generated with Claude Code

wavekat-eason and others added 26 commits April 17, 2026 19:45
Plan for adding Qwen3-TTS 0.6B Base voice clone (ref audio + ref text)
as a sibling backend to the existing 1.7B VoiceDesign pipeline.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Local Claude Code state (settings.local.json, worktrees/) shouldn't be tracked.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
API shape, encoder precision, and HF repo policy moved from open questions
to confirmed decisions.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Adds two new export scripts plus Makefile targets for the 0.6B Base voice
clone variant:

- export_speaker_encoder.py: ECAPA-TDNN, mel (B,T,128) -> embed (B,1024)
- export_tokenizer_encoder.py: Mimi encoder, waveform (B,T) -> codes (B,16,F)

Talker / code predictor / vocoder / embeddings reuse the existing config-driven
scripts; the new clone-* Makefile targets pin them to the 0.6B model id and a
separate output directory so 1.7B and 0.6B builds don't overlap.

Encoders ship FP32 only (locked in plan); quantize_int4 is unchanged.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Remove help-clone; all clone-* targets already appear in make help via the
grep pattern. Add CLONE_MODEL_ID/CLONE_OUTPUT_DIR to the variables section.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Mimi encoder's conv padding calls .item() which creates data-dependent
guards incompatible with torch.export/dynamo. Switch to the legacy JIT
tracer with a fixed canonical size (10s @ 24kHz = 240k samples).

Host code zero-pads reference audio to 240k samples before inference and
trims output codes based on the original sample count.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Mimi's encoder_transformer uses create_causal_mask with torch.vmap,
which is incompatible with JIT tracing (same issue as the talker).
Apply the existing mask_patch before tracing.

Also add [N/6] step labels to clone-export for clearer progress.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The Mimi model imports create_causal_mask directly into its local
namespace (`from transformers.masking_utils import create_causal_mask`),
so the existing module-level patch doesn't reach it. Add an explicit
patch of `transformers.models.mimi.modeling_mimi.create_causal_mask`.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
generate_clone_onnx.py chains all 6 ONNX models to produce cloned speech
from a reference audio + transcript. No PyTorch at inference time.

Pipeline: ref audio → tokenizer encoder (codes) + speaker encoder (mel →
embed) → ICL prefill → talker decode loop → code predictor → vocoder
→ trim ref portion → output WAV.

Also adds `make clone-generate` target.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Generate a 7.4s reference clip via macOS TTS (say) at 24kHz mono.
Stored at tools/qwen3-tts-onnx/fixtures/ref_clone.wav with gitignore
exceptions in both root and tools-level .gitignore files.

Makefile clone-generate now defaults to this fixture.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
make clone-fixture uses macOS say + librosa to regenerate
fixtures/ref_clone.wav from CLONE_REF_TEXT.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Replace macOS say-generated fixture with one produced by our own 1.7B
VoiceDesign backend (Apache-2.0 licensed Qwen3-TTS output). The
clone-fixture Makefile target now invokes cargo run --example synthesize.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Use "Speak in a warm and friendly female voice with natural intonation"
instead of the flat "Speak naturally and clearly" for a more natural
sounding reference clip.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Reference fixture now says "Give every small business the voice of a
big one." — the WaveKat tagline, consistent with all other examples.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Regenerate ref_clone.wav with two sentences (6.95s, well above the 3s
minimum for voice cloning) using FP32 precision for best quality.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Match how the committed fixture was actually generated.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Adds sibling Qwen3TtsClone struct with full ONNX pipeline:
mel spectrogram, speaker encoder, tokenizer encoder, ICL prefill,
talker decode loop, code predictor, and vocoder with ref-code trim.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The speaker_encoder.onnx and tokenizer_encoder.onnx use external
.data files for weights, but these were missing from the download
manifest. Also apply prepare_onnx_dir to the model root so ORT
can resolve HF Hub symlinks for the encoder sessions.

- Add .onnx.data entries to CLONE_SHARED_FILES in download.rs
- Skip directories in prepare_onnx_dir (safe for model root)
- Load encoders from symlink-resolved root directory
- Update voice clone doc to reflect implemented status
- Add fixtures/README.md documenting ref_clone.wav

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Load ref.wav before model in clone example to fail early, add ref.wav
requirement note, and print output info after synthesis in all examples.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@wavekat-eason
wavekat-eason merged commit c8488f6 into main Apr 17, 2026
5 checks passed
@wavekat-eason
wavekat-eason deleted the feat/qwen3-tts-0.6b-voice-clone branch April 17, 2026 09:58
@github-actions github-actions Bot mentioned this pull request Apr 17, 2026
wavekat-eason pushed a commit that referenced this pull request Apr 17, 2026
## 🤖 New release

* `wavekat-tts`: 0.0.2 -> 0.0.3 (✓ API compatible changes)

<details><summary><i><b>Changelog</b></i></summary><p>

<blockquote>

##
[0.0.3](v0.0.2...v0.0.3)
- 2026-04-17

### Added

- Qwen3-TTS 0.6B voice clone
([#14](#14))

### Fixed

- auto-resample ref audio to 24 kHz
([#15](#15))

### Other

- add model weights section
([#12](#12))
</blockquote>


</p></details>

---
This PR was generated with
[release-plz](https://github.com/release-plz/release-plz/).

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Great Project, do you plan to add the VoiceClone model Qwen3-TTS-Base?

1 participant